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!• INTROD'UCTION 



The Civil Service Reform Act (CSRA) of 1978 requires that each Govern- 
ment agency develop a performance-based employee appraisal system. The 
system must permit accurate assessment on the basis of objective criteria. 
Within the parameters of the law, the Air Force Human Resources Laboratory 
(AFHRL) has developed a new approach to evaluating the job performance of 
more than 200,000 Air Force civilian employees. The system is called the 
Job Performance Appraisal System (JPAS). 

This system involves breaking each job down into its major components, 
much the same as is done in preparing a position description. Task Analysis 
is used to make a list of teey work behaviors or practices essential to job 
performance. Standards of performance are set for each item on the list, 
and employees are then rat^d against these behaviors. In order to enhance 
employee participation and commitment to performance improvement, the 
employee and supervisor are urged to jointly participate in the process. 

The JPAS has three main components: (a) a work plan which sets forth 
the Job Performance Elements (job behaviors) and associated performance 
standards; (b) a system for reviewing the employee's progress in meeting the 
standards; and (c) a system for rating the employee ' s performance relative 
to the standards. While any rating system is evaluative in nature, the 
strength of this r.j>proach is that the criteria against which the employee is 
assessed are very objective* That is, there should be little or no argument 
about whether a performance standard was met. 

The JPAS will be used for all civilian employees not identified as mem- 
bers, of the Federal Senior Executive Service or General Manager employees. 
Therefore, it will be used for all Federal Wage System (FWS) and General 
Schedule (GS) employees. This means that the same system will be applied 
across a wide range of jobs at all locations where civilians are employed. 
Figure 1 contains a flow chart detailing the procedures associated with 
JPAS . 

The major objective of this work was to determine experimentally the 
most efficient, effective, and transportable approach to mediated training 
in the specific JPAS skills and knowledges required to effectively 
participate in the system. Elements contributing to the complexity of the 
problem were as follows: 

1. Size and heterogeneity of the population. 

2. Variability of media resources and classroom facilities at Air 
Force Bases where the training is to be conducted, 

3. Unknown capabilities of personnel who would be responsible for 
delivering training. 

4* Resistance to the CSEIA. Research on development of JPAS indicated 
that there is considerable resistance to acceptance of the new 
approach as mandated by the law. Therefore, beyond teaching people 
to use the system, it was hoped that the training would serve as a 
means of making JPAS more acceptable to users. 
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JOB l^EKFOHMANCE APPRAISAL SYSIEM QPAS) 



Yes 



Yes 



Yes 




12 



1. Supervisor informs employee of Job Performance 
Appraisal System, 

2. Work Plan Mrcling: 

— Iiiiicntify Jol) IVrformance Etemenla (JPEs) 

— Uelcrminc Criiical JPF.s 

• — Assign Relative Importance Points 

— Set Performance Standards 

3. Supervisor signs and s<Mids Wofk Plan to reviewer. 

4. Iteviewer checks Work Plan. 

5. Indicative of job? 

0. M eets organizational requircmcnis? 
7. Written in proper format? 

Hcviewer signs and rriurns Work Plan to supervisor 
for frmploycii's signature. 

9. Employee and supervisor retain a copy of the Wurk 
Plan. 

10. Supervisor completes and signs appraisal and forwards 
it to reviewer. 

1 L Reviewer checks and signs appraisal. 

12. .\ppraisal riMurntrd to supervisor to obtain employees 
signature*. 

13. Heturnod to supervisor for reaccomplishment. 

14. Returned to supervisor for reaccomplishment. 



Figure 1, Flowchart of JPAS Activities 



II. LITEEIATUR£ REVIEW 



The results of 50 years of comparative research indicate that any 
medium or media combination can teach about as well as "traditional instruc- 
tion." This is a meager information yield., given the thousands of research 
studies that have been conducted. Several reviewers (Allen, 1971; Clark, 
1975; Jamison, Suppeo, & Wells, 1974; Solomon & Clark, 1977) agree that the 
most usual kind of study, in which the scores on test X for group A who 
received treatment A are compared to the scores of group B who receivad 
treatment B, simply cannot provide information on those aspects of a treat- 
ment which are particularly important instructionally . In general, the more 
traditionally "well-controlled" a study is, the more likely it is to show no 
difference between treatments and, et^ually important, the more likely it is 
that the medium employed in the study was not used to its best advantage. 

Two consistent findings are that (a) active learner response during 
instruction tends to improve learning (Campeau, 1974) > and (b) presenting 
the learner with knowledge of the correct answer after a response also tends 
to increase learning (Reid, 1971). Thus, while the review of the literature 
narrowed the field of possible training methods to those which are active 
and performance-based, a clear picture of the methods which would be most 
effective did not enrerge. 



Ill, RESEARCH PLAN 



Instructional Materials 

Two prototype training courses were developed, differing on the type of 
active responding, feedback employed, and the mode by which information was 
presented. Each was designed to be of practical use to the student and to 
develop an attitude of acceptance toward the system. Each covered the same 
information and had the same objectives. The topics included in the 



instruction are: 




Unit 


I . 


An Overview of the Air Force Civilian Appraisal System 


Unit 


II. 


Job Performance Elements (JPEs) 


Unit 


III. 


Critical JPEs 


Unit 


IV. 


Performance Standards 


Unit 


V. 


Priority Weighting 


Unit 


VI, 


Work Plan Form 


Unit 


VII. 


Effective Communication and the Work Plan Meeting 


Unit 


Vllt. 


Role of the Reviewer in the Appraisal System 


Unit 


IX. 


Periodic Review 


Unit 


X. 


JPAS Ratings 


Unit 


XI. 


Post-Evaluation Process 



Prototype A was instructor-based. That is , an instructor was respon- 
sible for conducting lectures, demonstrations and group discussions. 
Feedback on exercises was given directly by the instructor or through 
instructor-led discussions. It was designed to allow opportunities for 



active discussion and inquiry about the system. This dynamic process 
addressed the problems associated with acceptance of the system. The 
materials were detailed instructor lesson plans with overheads and a student 
Workbook containing exercises and case studies. Instruction took place in a 
conventional classroom with information presented by the instructor, who 
also led the class through a series of application exercises contained in 
the Workbook* 

Prototype B was self-instructional. The complexity of the problem 
necessitated a solution which assured easy implement at ioa and avoided the 
requirement for locally available media equipment . Consequently, a pro-- 
grammed self-instructional text was designed. The same information used in 
the instructor-led course was contained in the programmed text. The 
material consisted of a Resource Book and a Workbook. Students worked at 
their own pace reading the Resource Book and applying what they read to 
exercises in the Workbook. 



Dependent Variables 

A performance test was developed to assess how well each prototype 
prepared students to use the JPAS . Measures were also developed to evaluate 
students' attitudes toward the JPAS as a function of the training prototype 
used and toward the instructional prototypes themselves* 

Performance Test . A two-part performance test was developed in which 
students had to apply the JPAS for three different job holders (a) Job 1, a 
service station attendant, (b) Job 2, a grocery clerk, and (c) Job 3, a 
meter reader. . These jobs are obviously not Air Force positions, but they 
were selected for their general familiarity. Since no special expertise is 
required to understand the job tasks, it was assumed that the content of the 
tests was not biased. 

In Part 1 of ..the performance test, the task was to write job perfor- 
mance elements (JPEs) and performance standards for each job. The materials 
used for this task were (a) a position description, (b) an audiovisual 
presentation, and (c) transcripts of interviews with both the employee and 
supervisor... In Part 2 the task was to use the rating system to evaluate 
each job holder's performance relative to the standards. The materials used 
here were a hypothetical work plan and a sample rating diary. Since both 
tasks were performed for each of the three jobs, there were a total of six 
subtests. The test was used in a pilot . study at AFHRL to determine: 

(a) if the instruments could be easily administered, and 
' (b) if the materials were sufficient in content to permit performance 
of the required task. 

An in-house tryout was conducted to verify usability of the materials. 
Five AFHRL staff members participated in the tryout and were selected 
because of their familiarity with the JPAS. Results of the tryout indicated 
that the package of materials was easy to administer and sufficient in 
content to permit development of a feasible work plan. Only minor revisions 
were required. 



Checklists were developed to be used in scoring the work plans. Inter- 
rater reliability coefficients (Pearson r) for three persons vrtio scored the 
results of the tryout were .98, .93, and .92. 

Attitude Scales , An attitude instrument consisting of a semantic 
differential scale was developed to ascertain the acceptability of the new 
System. Students were asked to evaluate the training programs also. 
Likert-type instruments were developed to assess student attitudes toward 
the prototypes. They were administered to each group after training. On a 
scale from 1 to 7 , with 1 being the most favorable, students were asked to 
place an "X" on the line at the point which reflected how they felt about 
the course on 16 dimensions. The scales were directed at assessing the 
students' general interest in the course and the perceived effectiveness of 
specific course components. 

Copies of all instruments for the performance test and attitude scales 
are presented in the appendix. With minor variations, explained later, the 
same instruments were used in each stage . 

General Procedures 

The research to determine the more effective approach was conducted in 
two stages;. In. stage one, training designs using the two prototypes were 
tested experimentally at Wright-Patterson AFB. As a result of the findings, 
changes aimed at enhancing the teaching effectiveness were made in the 
training design. Then the modified prototypes were compared in the second 
stage at Randolph AFB. Thus, two research studies were carried out and are 
described separately in this report. 



IV. STAGE ONE COMPARISONS 



Independent Variables 

Prototype A was broken out into two approaches according to the 
instructor's knowledge about the JPAS . Two types of instructors were 
selected to administer Prototype A. 

1. Instructors who were unfamiliar with JPAS, but typically would be 
expected to conduct the training at Air Force bases because of 
their availability* They were labeled "naive." 

2. Instructors who were subject matter experts (i.e., highly knowl- 
edgeable about JPAS because of their long-term involvement with the 
system) and who were , therefore , assumed to be better at conveying 
accurate information about the system than the naive instructors. 
The knowledgeable instructors were labeled Subject Matter Experts 
(SMEs). 
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Consequently, Che experimental treatments were as follows: 

Group 1. Students taught by naive (non-SME) instructors using the 
ins true tor-based materials . 



Group 2. Students taught by SMEs using the instructor-based materials. 

Group 3, Students using the self-instructional materials. 

The research hypothesis was that the performance of students in using 

the course materias and attitude of the students toward both the JPAS and 

the training course itself would be affected differentially by the type of 
training received . 



Procedures 



Instructor Orientation . In order to learn about the JPAS, the naive 
instructors worked through the self-instructional course. The lesson plans 
for conducting the training were given to them one week in advance of the 
course. They were told to review the materials as much as needed. Finally, 
the instructors participated in a question and answer session with a JPAS 
expert . 



No training was offered on classroom presentation methods. Both the 
SMEs and the naive instructors had had limited teaching experience. In this 
respect, they were typical of the kinds of personnel who might be expected 
to teach a JPAS course. 



Administration of Training Course . The Stage One field test was 
conducted at Wright-Patterson AFB. Instructor-based sessions were completed 
in two days (16 hours) of training. Self-instructional students worked at 
their own rate* The dependent measure was administered to all groups 
simultaneously on day three. 



Subjects 



The field test population consisted of 141 persons randomly sampled 
from the upper GS levels (a majority were GS 10 or above), some of whom were 
supervisors. The sample contained no FWS or union personnel. The subjects 
included engineers, accountants, contract specialists, and others in the 
professional, technical, and managerial categories. 

The field test population was assigned to three experimental treat- 
ments. Group 1 consisted of two classes led by "naive" instructors (n-42); 
Group 2 consisted of two classes led by SMEs (n=39); and Group 3 was the 
self-instructional (Sl) group (n-60) . Population figures represent the 
maximum number of students for which data are available. 
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Data Analysis 



Data were analyzed using the Analysis of Variance (ANOVA) for each of 
the six performance subtests and attitude scale. When significant differ- 
ences were found, the t-test for differences among several means was used to 
identify the source of the difference • The confidence level was set at .05. 

Due to student attrition, which was unrelated to the treatment, the 
total Ns for the dependent variable subtests varied. Reasons for different 
Ns in Stage One were: 

- Participation was voluntary. Students could choose not to attend 
the entire testing session. 

Student fatigue. Studencs failed to complete all parts of the 

test. This was especially true of Part 2 of the dependent 

variable. Notice that Job 3 of Part 2 has the smallest N (123). 

This was the last task to be completed in all cases. 

- Weather conditions. Icy roads caused several students to be late 
for the test. Thus, data were unavailable for those jobs 
administered early on the test day. 

Performance Test Results . Significant differences were found on five 
of the six subtests. This supports the hypothesis of differential effects 
on performance as a function of training design. Tables 1 and 2 present the 
ANOVA results for each comparison and Table 3, the means and standard 
deviations . 



Table 1. Stage One ANOVA Results for Performance Subtests 



Job 1: Part 1 Performance Standards 



Source 


SS 


df 


ms 


F 


P 


Between Groups 


144 


2 


72.0 


7 .32 


< 


.001 


Within Groups 


1,337 


136 


9.83 








Total 


1,481 


138 










Job 2: Part 1 Performance Standards 


Source 


SS 


df 


ms 


F 


P 


Between Groups 


65 


2 


32.59 


3.34 


< 


.001 


within Groups 


1,298 


133 


10.39 








Total 


1,363 


135 










Job 3: Part 1 Performance Standards 




Source 


SS 


df 


ms 


F 


P 


Between Groups 


120 


2 


60.0 


5.20 


• < 


.01 


Within Groups 


1,593 


138 


11.54 








Total 


1,713 


140 
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Table 2. Stage Two ANOVA Results for Performance Subtests 



Job 1: Part 2 Rating Performance 



1 Source 


SS 


df 




ms 




F 


P 


1 Between Groups 1 


182 1 


2 1 




90.75 1 




5.78 1 


<.005 i 


1 Within Groups 1 


2,120 1 


135 1 




15.70 1 




1 


- ■ 1 


1 Total 1 


2,301 1 


137 1 




1 




-- 1 


1 




Job 2: Part 2 Rating 


Performance 






1 Source 


SS 


df 


ms 




F 




1 Between Groups 


16 


2 1 




8.12 




.47 


NS 1 


1 Within Groups 


2,297 


136 




17.14 




- 


1 


1 Total 


2,313 


1 138 




-- 




- 


1 - 1 




Job 3: Part 2 Rating 


Performance 






1 Source 


SS 


df 


ms 




F 


P 


1 Between Groups 


1 84 


1 2 


1 


42.10 


1 


5.38 


1 <.01 


1 within Groups 


1 939 


1 120 


1 


7.83 


1 




i - 


i Total 


1 1,023 


1 122 


1 




1 







The t-test comparisons to determine which means differed significantly 
revealed that naive instructors had a detrimental effect on performance. 
See Tables 4 and 5 for t-test results. Students trained by SMEs performed 
significantly better than naive instructors' students on all conditions 
where a significant difference was found. Students using the self- 
instructional materials performed significantly better than students taught 
by naive instructors on four of the subtests. 

In comparing the performance of the self-instructional students with 
students led by SMEs, no significant difference on the six subtests was 
found. The data indicate that students working alone do as well as students 
led by SMEs. 
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Table 3. Stage One Group Means and Standard Deviations 

on the Performance Test 



Part 1. Rating Performance Standards 

Job 1 Job 2 Job 3 



X SD X SD X SD 

•Group 1 

(Naive) 15.02 3.67 16.24 2.83 14.90 3.41 
Group 2 

(SME-Instructor 

based) 17.42 2.69 17.92 3.60 17.36 3.24 
Group 3 

(Self-Instr.) 17.05 2.97 17.60 3.75 16.03 3.41 



Scoring of Performance Standards: 
5 points given for JPEs. 

1 point given for identification of a critical JPE. 

13 points given for performance standards . 

3 points given for assigning relative importance points. 

Total possible score: 22 points 



Part 2. Rating Performance 
Job 1 Job 2 Job 3 



X SD X SD X SD 



12.68 4.37 13.15 4.28 10.79 2.87 



15.50 3.66 13.89 3.76 12.17 2.54 



14.90 3.85 13.92 4.28 12.80 2.91 



Scoring of Rating Task 

7 points given for correctly using rating form. 
2 points given for each correct standard's rating. 
Total score Job 1-25 points 
Total score Job 2-27 points 
Total score Job 3-21 points 



Group 1 
(Naive) 

Group 2 
(Knowl.) 

Group 3 
(Self-In. ) 
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t-Test Results 



The t-test was used to determine which specific means differed 
significantly from each other after an F-ratio was found. In all instances, 
if the difference between any two means is larger than the critical 
difference, then the means are assumed to be significantly different. The 
critical difference is the product of the standard error of the difference 
among means and the t-value for the appropriate degrees of freedom and the 
alpha level of significance. 



Table 4. Stage One: Part I Performance Standards Test Results 

t-test for Job 1 

_Group 2 JSroup 3 Critical diff. 

X = 17.42 X = 17.05 (at ,05 level) 

Group 1: X= 15.02 2.40* 2.03* 1.30 
Group 2: X = 17,42 ~ ^7 1.30 



t-test for Job 2 

_Group 2 jGroup 3 Critical diff. 

X = 17.92 X = 17.60 (at ,05 level) 

Group 1: X = 16.24 1.68* 1.36* 1,33 

Group 2: X = 17.92 ~ ,32 1.33 



t-test for Job 3 

_Group 2 JGroup 3 Critical diff. 

X = 17.36 X = 16.03 (at ,05 level) 

Group 1: X= 14.90 2,46* 1.13 1,41 
Group 2: X = 17,36 ~ 1.33 1,41 



Table 5. t-Test for Stage Che: 
Part 2. Rating Performance Results 

t-test - Job 1 

_Group 2 JSroup 3 Critical diff. 

X = 15,50 X = 14.90 (at .05 level) 

Group 1: X= 12.68 2,82 2,22 1.66 
Group 2: X = 15.50 — ^60 1,66 



t-test - Job 3 

jGroup 2 JSroup 3 Critical diff. 

X = 12.17 X 12.80 (at ,05 level) 

Group 1: X = 10.79 1-38 2.01 1.26 
Group 2: X = 12.17 ^^3 1.26 
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At titude Toward System Results , The type of training received had no 
effec"t on students ' attitude toward the system. Means and standard 
deviations are presented in Table 6. On a scale from 1 to 7, with 7 being 
most favorable, the means were .4.46, 3.86, and 4.01, for the three groups 
respectively. Table 7 presents the ANOVA results. Because there is no 
significant difference, the hypothesis that training designs would have a 
differential effect on attitude toward the JPAS was rejected. 







Table 6. Groups Means 
Attitude 


and Standard Deviations 
Toward JPAS 


for 






N 


X 


SD 


Group 
Group 
Group 


1 
2 
3 


42 
39 

54 


4.46 
3.86 
4.01 


1 .67 
1 .29 
1.24 



Table 7. Stage One ANOVA Results for Attitude Toward JPAS 



Source 


SS 


df 


ms 


F 


P 


Between Groups 


8.71 


2 


4.36 


2.87 


NS 


Within Groups 


201.29 


132 


1.52 






Total 


210 


134 









Attitude Toward Training Results . Table 8 presents the means and 
standard deviations for each group. A significant difference was found in 
the student evaluation of training. Table 9 presents-the ANOVA results. 



Table 8. Stage One: Group Means and Standard Deviations 
for Attitude Toward Training 



Group 


N 


X 


SD 


1 - Naive Instr. 


37 


3.55 


.97 


2 - SME Instr. 


38 


3.22 


.82 


3 - Self-Instr. 


60 


3.89 


.97 



Table 9. Stage One: ANOVA Results for Attitude Toward Training 



Source 


SS 


df 


ms 


F 


P 


Between Groups 


10.69 


2 


5.34 


6.07 


<.005 


Within Groups 


115.57 


132 


.88 






Total 


126.26 


134 
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The t-test comparisons showed that students in Group 2 (SME) had a 
more favorable perception of the training than did the students in Group 3 
(self-instructional). See Table 10. Significant differences for naive 
ins tructor^led groups and the self-instructional group were not found. 



Table 10. 


Stage One: t-Test 


for Attitude 


Toward Training 








Critical 




Group 2 


Group 3 


Difference 




X = 3.22 


X = 3.89 


(at .05 Level) 


Group 1: X = 3.55 


.33 


.34 


.40 


Group 2: X = 3.22 




.67* 


.40 



The hypothesis that training prototype affects student attitude toward 
the course itself was confirmed. Students seem to prefer a course conducted 
by an SME over a self-instructional package (p<.005). This preference is 
not obtained when the instructor is not an SME. 



V. STAGE TWO COMPARISONS 



Independent Variables 

The prototypes evaluated in this stage were enhanced versions of those 
tested in Stage One. Specific enhancement strategies were as follows: 

1. The instructor-based and self-instructional prototypes were revised 
to overcome learning difficulties common to both. In addition, the 
JPAS rating procedures had been changed by AFHRL in the interim, 
and the course was revised to accommodate the new procedures. 

2. The naive instructor training design was eliminated on the basis of 
Stage One results. 

3. The SME-led training design was modified to add preparation on 
teaching skills and a more in-depth explanation of the JPAS to the 
instructors . All instructors in Stage One had expressed some 
anxiety about their delivery skills. Therefore, a one-half day 
training session was provided to allow practice and feedback on 
presentation methods and group facilitating. The hypothesis was 
that assistance on teaching skills would result in better student 
performance and attitude. 

4. A one-half day small group exercise on applying JPAS to a job was 
developed to be used after students had completed the self- 
instructional package. The hypothesis was that the addition of an 
opportunity for group discussion would result in better student 
performance and attitude. 
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Consequently, the experimental treatments were: 

Group 1. Students using the basic self -instructional package. 

Group 2. Students using the self-instructional package with a one-half 
day group application exercise. 

Group 3. Students taught by SMEs who had participated in an instructor 
training session and used the instructor-based package. 

The general research hypothesis was that performance and attitude of 
the students would be differentially affected by the type of training 
received . 

Dependent Variables 

Dependent measures were the same two-part performance test and the 
attitude scales used in Stage One. However, to reduce testing time, the 
performance test consisted of applying the JPAS to two jobs only: the 
service station attendant and the meter reader. Since the JPAS rating 
procedures had changed, a new scoring mechanism was devised for Part 2 of 
the performance test. 



Procedures 

The one-half day training session for instructors was conducted the day 
before the sessions began. The field test was conducted at Randolph AFB . 
Two days were allotted for instruction, followed by test administration on 
the third day. 

Subj ects 

The subjects wire 92 workers and supervisors randomly drawn from the 
lower GS levels and assigned to the experimental treatments. Subjects were 
mainly clerical and FVtS employees. Group 1 (n=29) received the self- 
instructional training. Group 2 (n=18) received the self-instructional 
training and group application exercise, and' Group 3 (n=45) received the SME 
training. Population figures represent the maximum number of students for 
vNich data are available. 



Data Analysis 

Data were analyzed using the ANOVA for each of the four subtests and 
the attitude scales. When significant differences were found, the t-test 
for differences among several means was used to identify the source of the 
differences. The confidence level was set at .05. 

Due to student attrition, which was non- treatment related, the total 
N's for the dependent variable components vary. 



ERIC 1^ 2 1 



Performance Test Results , Performance test results showed no signifi- 
cant differences in the ability of students to use the JPAS. The 
enhancements of each prototype had no effect on learning compared to the 
basic self-instructional prototype. Therefore, the research hypothesis of 
differential effects was rejected. As in Stage One, the self-instructional 
course and the SME-led course were equally effective. Table 11 presents 
the ANOVA results, and Table 12 the means and standard deviations. 



Table 11. Stage Two ANOVA Results for Performance 



Job 1 : Part 1 Performance Standards 



Source 


SS 


df 


ms F 


P 


Between Groups j 9.78 
Within Groups | 785.81 
Total 1 795.59 


2 1 
83 

85 1 


4.89 1 
9.47 1 


.52 
- 


NS j 

1 
1 


Job 2: Part 1 Performance Standards 


1 Source 


SS 


df 


mv5 F p 


1 Between Groups 
1 Within Groups 
1 Total 


12.21 
980.16 
992.37 


2 1 6.10 1 .52 i NS 1 
83 1 11.81 j — 1 — 1 
85 1 - 1 - 1 - [ 


Job 1: Part 2 Performance Standards 


Source 


SS 


df 


ms 


F 




1 Between Groups 
1 Within Groups 
1 Total 


18.24 1 2 1 9.12 
5,036.76 1 81 1 62.18 
5,055.00 i 83 i 


.15 


1 HS 1 

1 - 1 
1 - 1 


Job 2: Part 2 Performance Standards 


Source 


SS 


df 


ms 


F 


P 


1 Between Groups 
1 within Groups 
1 Total 


77.22 
7,859.58 
7,936.8 


2 
77 

1 79 


1 38.61 1 .38 1 NS 1 
1 .102.07 1 — 1 -- 1 
1 - 1 -- 1 - 1 
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Table 12. Stage Two Group Means and Standard Deviations 

on the Performance Test 



Part 1. 


Rating Performance 
Job 1 


Standards 

Job 2 






X 


SD 


X 


SD 


Group 1 

(Self-Instructional) 


15.41 


3.13 


15.86 


3.82 


Group 2 

(Sel f— instruct ional 
plus group 
application 


16.29 


3.60 


17.00 


4.07 


Group 3 

(SME-Ins true tor) 


16.05 


2.85 


16.27 


2,94 



Scoring of Performance Standards: 
5 points given for JPEs . 

1 point given for identification of a critical JPE. 

13 points given for performance standards. 

3 points given for assigning relative importance -points. 

Total possible score: 22 points 



Part 2, Rating Performance 







Job 1 


Job 


2 




X 


SD 


X 


SD 


Group 1 

(Self-Instructional) 


24.85 


8.36 


13.38 


10.45 


Group 2 

(Self-Instructional 
plus group 
application) 


25.53 


8.13 


25.43 


11.26 


Group 3 

(Self-Instructor) 


25.90 


7.48 


25.50 


9.45 



Scoring of Rating Task 

7 points given for correctly using rating form. 
2 points given for each correct standard's rating. 
Total score Job 1-25 points 
Total score Job 2-27 points 
Total score Job 3-21 points 
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. Attitude Results . . As in Stage One, results showed that type of 
training had no effect on attitude toward JPAS . Table 13 presents the ANOVA 
results* Table 14 presents the means and standard deviations. 



Table .13. Stage Two ANOVA Results for Attitude Toward the JPAS 



Source 


* SS 


df 


ms 


F 


P 


Between Groups 


3.26 


2 


1.63 


1.26 


NS 


Within Groups 


108.01 


84 


1.29 






Total 


111.27 


86 









Table 14. Group Means and Standard Deviations for 
Attitude Toward JPAS 





N 


X 


SD 


Group 1 


29 


4.79 


i.25 


Group 2 


15 


4.42 


.99 


Group 3 


43 


4.37 


l.!0 



The evaluation of training results revealed that students pr<eferi:ed the 
SME-led course over both of the other types of training courses adminis^ 
tered. Table 15 presents the means and standard deviations (1— favoral?le, 
7=unfavorable) , Table 16 presents the ANOVA results. Table 17 shows t-test 
results. 



Table 15. Stage Two: Group Means and Standard Deviations 
for Attitude Toward Training 



Group 


N 


X 


SD 


1 - Self Instr. 


29 


3.06 


l.li 


2 - Self-Instr./ 








Small Group 


18 


3.33 


.79 


3 - SME Instr. 


45 


2.55 


.85 



Table 16. Stage Two ANOVA Results for Attitude Toward Training 



Source 


SS 


df 


ms 


F 


P 


Between Groups 


16.39 


2 


8.20 


8.37 


<.001 


Within Groups 


87.58 


89 


.98 






Total 


103.97 


91 
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Table 17. t-Test for Stage Two: Attitude Toward Training 



Here the Ns were disparate- That is, there were more than twice as many 
in the largest group as in the smallest group (Group 1 N was 18 while Group 3 
N was 45). Therefore, the standard error was computed separately for each 
comparison. The results are presented below. 

Group 2 vs. Group 1 (C diff. = .60) 

3.33 - 3.06 = .17 (non-significant) 

Group 2 vs. Group 3 (C diff. = .54) 

3.33 - 2.55 = .78 (significant) 

Group I vs. Group 3 (C diff. ~ .48) 

3.06 - 2.55 = .51 (significant) 



VI. GENERAL DISCUSSION AND CONCLUSIONS 

Naive Instructors vs. SMEs 

In and instructor-based system, subject matter expertise appears to be 
important in training people to use the JPAS. Instructors who were labled 
naive had a detrimental effect on learning despite their precourse orientation 
to the JPAS and the use of highly detailed lesson plans. Observations and 
post-hoc analysis of the naive instructors' behavior indicate that they were 
less able to field questions about the system and that they introduced their 
misconceptions about the system. Students' attitudes toward the training is 
congruent with the finding that the subject matter expertise is a significant 
factor, as students taught by SMEs were more favorably inclined toward the 
training than those taught by non-SMEs. 

Clearly, on the basis of performance and attitude toward the training, 
field test results showed that if an instructor-based program is used to teach 
the JPAS, the instructor should be an SME . 

The SMEs employed in- this study were drawn from the AFHRL at Brooks AFB 
and the contractor (Applied Science Associates) employees who participated in 
development of the course. All of these people had been working with the JPAS 
for several months (i.e., from 4 months to more than 1 year). Thus, their 
indoctrination was long and intensive. 

If an instructor-based system is selected, a training program which 
thoroughly indoctrinates instructors in the use of the JPAS must be 
developed. The self-instructional package used to train the naive 
instructors in the Wright-Patterson field test was obviously inadequate. Just 
how much more training and what type of instruction is necessary to produce 
results similar to those achieved in the field test remains an .^"'P^';^"! 
Question. One, approach to determining the nature of a JPAS ins true to.r_. 
tt lining package is., to perform an anal/sis of the tasks performed by personnel 
in the process of becoming SMEs. In this way, the experiences necessary to 
becoming an SME can be identified and training jfjL be designed to simulate 
these conditions. 
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Self-Instructional vs, SME-led Training 



There are no differences in student performance or attitude toward the 
system between students using the self-instructional package and those 
t.;ught by SMEs^, However, students' attitudes toward the training is more 
favorable when instruction is presented by an SME. While the difference is 
statistically significant , the difference of about one half a point (.51) 
must be evaluated in terms of practical implications. 

On a scale of one to seven, with one being the most favorable, the mean 
of the self-instructional group was 3.06, which is on the moderate to 
favorable side of the scale as is the SME-led group with a mean of 2.55. 
The evidence reveals no real dislike for the self-instructional approach, 
but clearly the SME-led training is preferred. But, since this slight 
preference for SME-led training does not translate into better performance 
and acceptance of the system, the importance of this difference is 
questionable. 

In comparing the practical benefits of the two prototypes, however, 
there are certain administrative considerations which do appear to make a 
difference in terms of time and resource allocations. 

Figure 2 shows the completion times for the students who participated 
in the self-instructional sessions in Stage One. At the end of the first 
day, 33 percent of the students turned in the instructional materials and 
returned to their job. By 10 a.m. of the second day (after 10 instructional 
hours) an additional 40 percent were ready to return to work. All of the 
students turned in the materials within 14 hours • The median completion 
time was nine hours* Obviously, all students in the group-paced, 
instructor-led training finished at the same time (after 16 hours). As 
shown in Figure 3, essentially the same effect was found in the Stage Two 
test at Randolph AFB. That is, 50 percent of the students were finished 
after nine hours, and everyone was finished after 11.5 hours. 

If time away from the job is a factor, clearly the self-instructional 
approach is preferable since students achieved the same performance level as 
with the instructor-based training, but were returned to work sooner. This 
is a significant factor when large numbers of personnel are being trained. 

The self -instructional package can be administered by almost anyjDne, 
whereas the ins true torr-led course requires an SME as the instructor ... As 
discussed above, all potential JPAS instructors should become experts in the 
system and become familiar with the detailed lesson plans for conducting the 
training. Once the course is underway, the level of effort for actual 
administration of the self-instructional package is lower for two reasons: 
(a) the self-instructional administrators finish as soon as the last student 
has turned in his materials, approximately 12 to 14 hours, while SME 

-'•This study did not take the reading ability into account. Subjects were 
drawn from the populations at the bases and randomly assigned to treatments. 
No measures of reading ability were given. Therefore, the effectiveness of 
these packages for individuals with low reading ability is not known* 
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Figure 3. Stage Two. Completion Times of Self-Instructional Students 
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instructors require 16 hours to administer the course, and (b) the self- 
instructional administrators can monitor 35 or more students with ease, 
while the SME instructors should, according to the course design, teach no 
more than 25 persons. 

Certainly, the demand on human resources for the instructor-led course 
is significant when thousands of persons must be trained. Therefore, 
administratively and in terms of cost-effectiveness, the self -instructional 
package is preferable. 
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APPENDIX 
MEASUREMENT DEVICES 
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Check One: 



Self-Instructional 
Instructor - led 
Date: 



Instructor' s name 



ATTITUDE TOWARD JPAS 



Think about the Job Performance Appraisal System process. Then for 
each pair of words listed below, put an "X" on the line point 
which reflects how you feel about „it. You should record an *'X" for 
each pair of words. 



JOB PERFORMANCE EVALUATION SYSTEM 



Bad 
Complete 
Unsuccessful 
Unimportant 
Worthless 
Satisfactory 
Negative 
Wise 
Strong 
Ea s y 
Dimib 
Timely 
Wrong 
Interesting 
Pract ical 
Useless 



incomplete 

S^^cessful 

important 

valuable 

unsatisfactory 

positive 

Fooiish 

Weak 

Hard 

smart 

untimely 

Right 

pull 

impractical 
useful 
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GROUP // 



Evaluation of the Training 
Self -Instructional 



This evaluation is intended to find out about your attitude toward 

the course . Read each item and place an "X" in the line at the point 

which reflects how you feel about it. You should record an "X" for each 
pair of words. 



How did the course compare 


to other training 


you have taken? 


Good 1 :2 :3 


:4 


:5 :6 


:7 : Bad 


How interesting was the course in 


general? 




Interesting 1 :2 :3 




:5 :6 


:7 : Boring 


How often would you like to 


have 


courses like 


this one again? 


Frequently 1 :2 :3 


:A 


: 5 ; o 


:7 : Rarely 


How well did the course prepare you to apply 
JPAS system on the job? 


the elements of the 


a. Job Performance Elements? 






Very Well 1 :2 :3 


:4 


:5 :6 


:7 : Very poorly 


b. Critical Elements? 








Very Well 1 :2 :3 


:A 


:5 :6 


: 7 : Very poorly 


c. Performan'ce Standards? 








Very Well 1 :2 :3 


:A 


:5 :6 


: 7 : Very poorly 


d. Priority Weights? 








Very Well 1 :2 ;3 




:5 :6 


: 7 : Very poorly 
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Self'-Inst ructlonal , continued 

5. Hov well did the course prepiare you to complete the forms used 
in the system? 

a. Work Plan Form? 

Very Well 1 :2 :3 :4 :6 :7 : Very poorly 

b. Quarterly Work Plan Progress Review? 

Very Well 1 :2 :3 :4 :5 :6 :7 : Very poorly 

c. JPAS Rating Form (yearly)? 

Very Well I :2 :3 :A :5 :6 :7 : Very poorly 

6. How effective were the following materials in helping you learn? 
a* Practical exercises in the workbook? 

Ef f ec t Ive 1 : 2 : 3 : A : 5 : 6 : 7 : Ineffective 

b. Case Studies in Appendix A of the Workbook? 

Ef f ec t ive 1 : 2 : 3 : A : 5 : 6 : 7 : Inef f ec t ive 

7. How often did you have questions that the manual did not address? 
Very often 1 :2 ; 3 ;4 :5 :6 :7 : Rarely 

8. How sufficient were the answers to the exercises? 

Sufficient 1 :2 :3 :A : 5 :6 : 7 : Insufficient 

9. How well did you like working independently? 

Very much 1 :2 :3 ;A :5 :6 :7 : Not at all 

10. How clear were the directions for the activities in the manual? 

Very Clear 1 :2 :3 :A :5 :6 :7 : Very Unclear 



^5 
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GROUP // 



Evaluation of the Training 
Group-Paced 



This evaluation is intended to find out about your attitude toward 
the course . Read each item and place an "X" in the line at the point 
which reflects how you feel about it. You should record an "X" for each 
pair of words. 

1. How did the course compare to other training you have taken? 

Good 1 :2 :3 :4 :5 :6 :7 : Bad 

2. How interesting was the course in general? 

Interesting 1 :2 :3 :4 :5 :6 :7 : Boring 

3. How often would you like to have courses like this one again? 
Frequently 1 : 2 : 3 : 4 : 5 : 6 : 7 : Rarely 

A. How weir did the course prepare you to apply the elements of the 
JPAS system on the job? 

a. Job Performance Elements? 

Very Well 1 :2 :3 :4 :5 :6 :7 : Very poorly 

b. Critical Elements? 

Very Well 1 :2 :3 :4 :5 ;6 :7 : Very poorly 

c. Performance Standards? 

Very Well 1 :2 :3 :4 :5 :6 :7 : Very poorly 

d. Priority Weights? 

Very Well 1 :2 :3 :4 :5 :6 :7 : Very poorly 
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Group-Paced , continued 

5. How well did the course prepare you to complete the forms used in 
the system? 

a. Work Plan Form? 

Very Well Ij :2 :3 :A :5 :6 :7 : Very Poorly 

b. Quarterly Work Plan Progress Review? 

Very Well 1 :2 :3 :A :5 :6 :7 : Very Poorly 

c. JPAS Rating Form (yearly)? 

Very Well 1 :2 :3 :A :5 :6 :7 : Very Poorly 

6. How effective were the following materials in helping you learn? 

a. Practical exercises in the workbook? 

Effective 1 :2 :3 :A :5 :6 : 7 : Ineffective 

b. Case Studies in Appendix A of the Workbook? 

Effective 1 :2 : 3 :A :5 :6 :7 : Ineffective 

7. How effective were the following in supporting the instruction? 
a. Instructor's presentation? 



Very 

Ineffective 1 :2 


:3 




:5 


:6 


:7 


Very 
: Effective 


b • Group interaction? 
Very 

Ineffective 1 :2 


:3 




:5 


:6 


:7 


Very 
: Effective 


c. Your participation? 
Very 

Ineffective 1 :2 


:3 


;A 


:5 


:6 


■ :7 


Very 
: Effective 


d. Audio-Visual aids? 
Very 

Ineffective 1 :2 


:3 . 




:5 


:6 


:7 


Very 
: Effective 
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